Rényi divergence and majorization
نویسندگان
چکیده
Rényi divergence is related to Rényi entropy much like information divergence (also called Kullback-Leibler divergence or relative entropy) is related to Shannon’s entropy, and comes up in many settings. It was introduced by Rényi as a measure of information that satisfies almost the same axioms as information divergence. We review the most important properties of Rényi divergence, including its relation to some other distances. We show how Rényi divergence appears when the theory of majorization is generalized from the finite to the continuous setting. Finally, Rényi divergence plays a role in analyzing the number of binary questions required to guess the values of a sequence of random variables.
منابع مشابه
Image Registration and Segmentation by Maximizing the Jensen-Rényi Divergence
Information theoretic measures provide quantitative entropic divergences between two probability distributions or data sets. In this paper, we analyze the theoretical properties of the Jensen-Rényi divergence which is defined between any arbitrary number of probability distributions. Using the theory of majorization, we derive its maximum value, and also some performance upper bounds in terms o...
متن کاملMajorization, Csiszár divergence and Zipf-Mandelbrot law
In this paper we show how the Shannon entropy is connected to the theory of majorization. They are both linked to the measure of disorder in a system. However, the theory of majorization usually gives stronger criteria than the entropic inequalities. We give some generalized results for majorization inequality using Csiszár f-divergence. This divergence, applied to some special convex functions...
متن کاملStatistical Functionals Consistent with a Weak Relative Majorization Ordering: Applications to the Mimimum Divergence Estimation
Most of the statistical estimation procedures are based on a quite simple principle: find the distribution that, within a certain class, is as similar as possible to the empirical distribution, obtained from the sample observations. This leads to the minimization of some statistical functionals, usually interpreted ad measures of distance or divergence between distributions. In this paper we st...
متن کاملVon Neumann entropy and majorization
We consider the properties of the Shannon entropy for two probability distributions which stand in the relationship of majorization. Then we give a generalization of a theorem due to Uhlmann, extending it to infinite dimensional Hilbert spaces. Finally we show that for any quantum channel Φ, one has S(Φ(ρ)) = S(ρ) for all quantum states ρ if and only if there exists an isometric operator V such...
متن کاملA Preferred Definition of Conditional Rényi Entropy
The Rényi entropy is a generalization of Shannon entropy to a one-parameter family of entropies. Tsallis entropy too is a generalization of Shannon entropy. The measure for Tsallis entropy is non-logarithmic. After the introduction of Shannon entropy , the conditional Shannon entropy was derived and its properties became known. Also, for Tsallis entropy, the conditional entropy was introduced a...
متن کامل